Skip to content

Support Unicode Gem path #493

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Support Unicode Gem path #493

wants to merge 1 commit into from
+5 −1

Conversation

@davispuh
Copy link

I'm using RVM and my $HOME is located at /home/Dāvis which means GEM path is at /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/

Currently this causes bootsnap to fail with

/home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `start_with?': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `block in stability'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `detect'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:124:in `stability'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:12:in `stable?'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/path.rb:60:in `entries_and_dirs'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:155:in `block (2 levels) in push_paths_locked'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:147:in `each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:147:in `block in push_paths_locked'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/store.rb:53:in `block in transaction'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/store.rb:52:in `synchronize'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/store.rb:52:in `transaction'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:146:in `push_paths_locked'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:119:in `block in push_paths'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:119:in `synchronize'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/cache.rb:119:in `push_paths'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/change_observer.rb:11:in `<<'

This is because Gem.path is at ["/home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0"] and it's UTF-8 encoded while sometimes path can be with ASCII-8BIT encoding.

Note that there is similar issue within Ruby aswell, see ruby/ruby#11958

@casperisfine
Copy link
Contributor

Could you provide some reproducible test case? I'm trying to reproduce your problem but even by creating some mix encoding paths, I'm not able to:

>> Dir["**/*.rb"]
=> ["t€st/\xC3.rb"]
>> Dir["**/*.rb"].map(&:encoding)
=> [#<Encoding:UTF-8>]

I suspect there's more to this problem, like a LOCALE issue perhaps.

@davispuh
Copy link
Author

I'm not exactly sure what causes to use ASCII-8BIT encoding but this example shows how require path is in different encoding

# encoding: ISO-8859-1

module Kernel
    alias_method :actual_require, :require

    def require(path)
       puts "requiring #{path} with #{path.encoding} encoding"
       actual_require(path)
    end
end

require 'tzinfo/data'
requiring tzinfo/data with ISO-8859-1 encoding
requiring tzinfo/data/version with UTF-8 encoding

My LANG env var is lv_LV.UTF-8 and I don't use any other locale than UTF-8.

@casperisfine
Copy link
Contributor

I'm not exactly sure what causes to use ASCII-8BIT encoding

Can you provide the entire backtrace then?

@davispuh
Copy link
Author

I'm not exactly sure what causes to use ASCII-8BIT encoding

Can you provide the entire backtrace then?

That is whole backtrace, for some reason it doesn't show anything more.

Basically this happens when I try to use Metasploit Framework msfconsole (https://github.com/rapid7/metasploit-framework)
but first issue I encounter is Ruby's ruby/ruby#11958 after fixing that I get this one.

Also note that most of requires work fine, only some deeply in loading when it tries to load tzinfo/data it fails.

Here's full backtrace without Ruby's fix

/home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:102:in `delete_prefix': incompatible character encodings: ASCII-8BIT and UTF-8 (Encoding::CompatibilityError)
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:102:in `warning?'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:71:in `block (2 levels) in replace_require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/zeitwerk-2.6.17/lib/zeitwerk/kernel.rb:34:in `require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/data_source.rb:149:in `create_default_data_source'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/data_source.rb:55:in `block in get'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/data_source.rb:54:in `synchronize'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/tzinfo-2.0.6/lib/tzinfo/data_source.rb:54:in `get'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/activesupport-7.0.8.4/lib/active_support/railtie.rb:88:in `block in <class:Railtie>'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/initializable.rb:32:in `instance_exec'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/initializable.rb:32:in `run'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/initializable.rb:61:in `block in run_initializers'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:231:in `block in tsort_each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:353:in `block (2 levels) in each_strongly_connected_component'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:434:in `each_strongly_connected_component_from'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:352:in `block in each_strongly_connected_component'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:350:in `each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:350:in `call'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:350:in `each_strongly_connected_component'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:229:in `tsort_each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/tsort.rb:208:in `tsort_each'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/initializable.rb:60:in `run_initializers'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/application.rb:372:in `initialize!'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/railtie.rb:226:in `public_send'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/railties-7.0.8.4/lib/rails/railtie.rb:226:in `method_missing'
        from /opt/metasploit/config/environment.rb:4:in `<top (required)>'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `block (2 levels) in replace_require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
        from /opt/metasploit/lib/msfenv.rb:28:in `<top (required)>'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/3.3.0/bundled_gems.rb:74:in `block (2 levels) in replace_require'
        from /home/Dāvis/.rvm/rubies/ruby-3.3.4/lib/ruby/gems/3.3.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
        from /opt/metasploit/msfconsole:21:in `<main>'

@casperisfine
Copy link
Contributor

I'm gonna wait to see how ruby/rubygems#8196 and ruby/ruby#11958 are resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@davispuh @casperisfine @rafaelfranca